Image collection with increased accuracy

ABSTRACT

Image collection is provided, e.g., a system includes a gaze point detection device for detecting the location of a gaze point of a user with respect to the user, and an image collection device for focusing and collecting a first image according to the location of the gaze point with respect to the user. The gaze point detection device comprises a fundus image collection module for collecting a second image of a fundus of a user, an adjustable lens module for adjusting an imaging parameter of an optical path between the fundus image collection module and an eye and an image processing module for calculating the location of a gaze point of the user with respect to the user according to the imaging parameter. The system and method of the present application can increase the focusing accuracy and the focusing speed of an image collection apparatus.

RELATED APPLICATION

The present application claims the priority of Chinese PatentApplication No. 201310419787.0, entitled “Image collection system andimage collection method” and filed on Sep. 16, 2013, which is herebyincorporated herein by reference in its entirety.

TECHNICAL FIELD

The present application relates to the technical field of imagecollection, and, in particular, to image collection that achievesgreater accuracy.

BACKGROUND

As the human-computer interaction technology develops increasingly, thehuman-computer interface more emphasizes the “human-centered” principle,so that a user can use various sensory channels to interact with acomputer apparatus in a most natural manner.

The sight line of a user reflects the interested object, destination anddemand of the user, having an input-output bidirectional characteristic.Gaze point detection makes it possible to extract information useful forhuman-computer interaction, so as to realize a natural, intuitive andeffective interaction, and therefore is applied to various aspects ofhuman-computer interaction more and more. For example, during thefocusing of an image collection apparatus, the interested object of auser can be determined by detecting a gaze point of the user to therebyhelp the image collection apparatus in focusing.

SUMMARY

The following presents a simplified summary in order to provide a basicunderstanding of some aspects or embodiments disclosed herein. Thissummary is not an extensive overview. It is intended to neither identifykey or critical elements nor delineate the scope of the aspects orembodiments disclosed. Its sole purpose is to present some concepts in asimplified form as a prelude to the more detailed description that ispresented later.

An example, non-limiting object of the present application is to providean image collection technology to increase the focusing accuracy of animage collection apparatus.

In a first example embodiment, the present application provides an imagecollection system, comprising:

a gaze point detection device for detecting the location of a gaze pointof a user with respect to the user; and

at least one image collection device for focusing and collecting atleast one first image according to the location of the gaze point withrespect to the user;

wherein the gaze point detection device comprises:

a fundus image collection module for collecting at least one secondimage of a fundus of a user;

at least one adjustable lens module for adjusting at least one imagingparameter of an optical path between the fundus image collection moduleand an eye of the user, until the fundus image collection module obtainsat least a subset of the at least one second image that satisfies atleast one defined clarity criterion; and

an image processing module configured to process at least the subset ofthe at least one second image that satisfies the at least one definedclarity criterion, obtain the at least one imaging parametercorresponding to at least the subset of the at least one second imagethat satisfies the at least one defined clarity criterion and an opticalaxis direction of the eye, and determine the location of the gaze pointof the user with respect to the user.

In a second example embodiment, the present application provides amethod, comprising:

detecting a location of a gaze point of a user with respect to the user;and

focusing and collecting at least one first image according to thelocation of the gaze point with respect to the user,

wherein the detecting comprises:

collecting at least one second image of a fundus of the user;

adjusting at least one imaging parameter of an optical path between acollection location of the at least one second image and an eye of theuser, until at least a subset of the at least one second image thatsatisfies at least one defined clarity criterion is obtained during thecollecting the at least one second image of the fundus of the user;

processing at least the subset of the at least one second image thatsatisfies the at least one defined clarity criterion;

obtaining the at least one imaging parameter corresponding to at leastthe subset of the at least one second image that satisfies the at leastone defined clarity criterion and an optical axis direction of the eye;and

determining the location of the gaze point of the user with respect tothe user.

In a third example embodiment, the present application provides acomputer-readable storage device, comprising at least one executableinstruction, which, in response to execution, causes a device comprisinga processor to perform operations, comprising:

detecting a location of a gaze point of a user with respect to the user;and

focusing and collecting at least one first image according to thelocation of the gaze point with respect to the user;

wherein the detecting the location of the gaze point comprises:

collecting at least one second image of a fundus of the user;

adjusting at least one imaging parameter of an optical path between acollection location of the at least one second image and an eye of theuser, until the at least one second image that satisfies at least onedefined clarity criterion is obtained during the collecting of the atleast one second image of the fundus of the user;

processing the at least one second image that satisfies the at least onedefined clarity criterion;

obtaining the at least one imaging parameter corresponding to the atleast one second image that satisfies the at least one defined claritycriterion and an optical axis direction of the eye; and

calculating the location of the gaze point of the user with respect tothe user.

In a fourth example embodiment, the present application provides animage collection system, characterized by comprising a processing deviceand a memory, wherein the memory stores at least one executableinstruction, the processing device is connected to the memory via acommunication bus, and when the image collection system executes, theprocessing device executes the at least one executable instructionstored in the memory, so that the image collection system executesoperations, comprising

detecting a location of a gaze point of a user with respect to the user;and

focusing and collecting at least one first image according to thelocation of the gaze point with respect to the user;

wherein the detecting the location of the gaze point comprises:

collecting at least one second image of a fundus of the user;

adjusting at least one imaging parameter of an optical path between acollection location of the at least one second image and an eye of theuser, until the at least one second image that satisfies at least onedefined clarity criterion is obtained during the collecting of the atleast one second image of the fundus of the user; and

processing the at least one second image that satisfies the at least onedefined clarity criterion;

obtaining the at least one imaging parameter corresponding to the atleast one second image that satisfies the at least one defined claritycriterion and an optical axis direction of the eye; and

calculating the location of the gaze point of the user with respect tothe user.

At least one technical solution of the embodiments of the presentapplication obtains the location of a gaze point of an eye of a user bydetecting the relative distance from the gaze point of the eye of theuser to the user and the gaze direction of the eye of the user, so as tocomplete the automatic focusing of an image collection device, with highaccuracy rate and rapid speed of focusing thereof;

at least one technical solution of the embodiments of the presentapplication adopts an electronic focal-length adjustable lens to help afundus image collection module acquire a clear image of a fundus of auser, so that the gaze point detection device has a compact structureand small volume, and is more suitable to be applied to a wearablenear-to-eye apparatus, making it more convenient to use an imagecollection system;

at least one technical solution of the embodiments of the presentapplication acquires the distance of a gaze point of an eye of a userwith respect to the user by shooting an image of the fundus itself ofthe user or projecting a light spot pattern to the fundus of the user,making the detection accuracy better;

and at least one technical solution of the embodiments of the presentapplication may comprise a plurality of cooperatively focusing imagecollection devices, which help a user obtain a desired image.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an example structural schematic block diagram of an imagecollection system of the embodiments of the present application;

FIG. 2 is an example application scene schematic diagram of an imagecollection system of the embodiments of the present application;

FIG. 3 is an example application scene schematic diagram of anotherimage collection system of the embodiments of the present application;

FIG. 4a is an example structural block diagram of a gaze point detectiondevice of an image collection system of the embodiments of the presentapplication;

FIG. 4b is an example schematic diagram of a light spot pattern used bya gaze point detection device of an image collection system of theembodiments of the present application;

FIG. 4c is an example schematic diagram of a fundus image with a lightspot pattern shot by a gaze point detection device of an imagecollection system of the embodiments of the present application;

FIG. 5a is an example optical path schematic diagram of eye imaging of agaze point detection device of an image collection system of theembodiments of the present application;

FIG. 5b is an example schematic diagram where a gaze point detectiondevice of an image collection system of the embodiments of the presentapplication obtains the location of a gaze point of an eye with respectto a user according to known imaging parameters of the system;

FIG. 6 is an example schematic diagram of a gaze point detection deviceof an image collection system of the embodiments of the presentapplication applied to a pair of spectacles;

FIG. 7 is an example schematic diagram of a gaze point detection deviceof another image collection system of the embodiments of the presentapplication applied to a pair of spectacles; and

FIG. 8 is an example step flowchart of an image collection method of theembodiments of the present application.

DETAILED DESCRIPTION

The method and device of the present application are described in detailhereinafter with reference to the drawings and embodiments.

As shown in FIG. 1, one or more of the embodiments of the presentapplication provide an image collection system 100, comprising:

a gaze point detection device 110 for detecting the location of a gazepoint of a user with respect to the user; and

at least one image collection device 120 for focusing and collecting atleast one first image according to the location of the gaze point withrespect to the user;

wherein the gaze point detection device 110 comprises:

a fundus image collection module 111 for collecting at least one secondimage of a fundus of a user;

an adjustable lens module 112 for adjusting at least one imagingparameter of an optical path between the fundus image collection module111 and an eye of the user, until the fundus image collection module 111obtains at least one second image that satisfies at least one definedclarity criterion; and

an image processing module 113 for processing the at least one secondimage that satisfies at least one defined clarity criterion, obtainingthe at least one imaging parameter of the optical path between thefundus image collection module 111 and the eye corresponding to the atleast one second image that satisfies at least one defined claritycriterion and the optical axis direction of the eye, and calculating thelocation of the gaze point of the user with respect to the user (forexample, the location thereof with respect to the eye of the user).

Here, the at least one image collection device 120 is used forcollecting an image of an object which a user intends to shoot (forexample, an object on which the user is fixing) to obtain at least onefirst image.

In this implementation, the at least one defined clarity criterion, forexample, can be a criterion commonly used by those skilled in the artfor judging whether an image is clear, such as resolution and so on.

Furthermore, the embodiments of the present application also provide acomputer-readable medium, comprising a computer-readable instructionupon the execution of which the following operations are performed, theoperations being as follows:

detecting the location of a gaze point of a user with respect to theuser; and

focusing and collecting at least one first image according to thelocation of the gaze point with respect to the user;

wherein the detecting the location of a gaze point of a user withrespect to the user comprises:

collecting at least one second image of a fundus of the user;

adjusting at least one imaging parameter of an optical path between thecollection location of the at least one second image and an eye of theuser, until at least one second image that satisfies at least onedefined clarity criterion is obtained during the collecting at least onesecond image of a fundus of the user; and

processing the at least one second image that satisfies at least onedefined clarity criterion, obtaining the at least one imaging parametercorresponding to the at least one second image that satisfies at leastone defined clarity criterion and the optical axis direction of the eye,and calculating the location of the gaze point of the user with respectto the user.

One or more embodiments of the present application obtain the opticalaxis direction of an eye of a user and the distance from a gaze point tothe eye by means of the gaze point detection device 110, thereby beingcapable of calculating out the location of the gaze point of the userwith respect to the user, and then focus for the collection of a firstimage according to the location of the gaze point, so that the imagecollection system 100 can rapidly and accurately focus and collect animage of the interested object of the user, enhancing the userexperience.

As shown in FIG. 2, in a possible implementation of the embodiments ofthe present application, the system further comprises a wearablenear-to-eye apparatus 210, the gaze point detection device 270 beingarranged on the wearable near-to-eye apparatus 210. In an implementationof the present application, the wearable near-to-eye apparatus can be apair of spectacles (including a pair of framed spectacles, contactlenses, goggles, etc.) and other devices which are easy to carry andconvenient to use. Especially for a user who has suffered fromrefractive error and other eye problems and needs to wear, for example,a pair of spectacles with correction of refractive error, the system ofthe present application can be realized directly on the pair ofspectacles with correction of refractive error, without bringing aboutan extra burden to the user. In other possible implementations of theembodiments of the present application, the system may also comprise,for example, other optical apparatuses used matching the eyes of a user,such as a helmet eyepiece, a front windshield for driving, etc., and thegaze point detection device can also be arranged on these opticalapparatuses. Of course, it can be known by those skilled in the art thatthe gaze point detection device itself can also form a wearablenear-to-eye apparatus, rather than has to be attached to otherapparatuses.

In a possible implementation of the embodiments of the presentapplication, the system comprises an image collection device 220, theimage collection device 220 also being arranged on the wearablenear-to-eye apparatus 210.

In this implementation, the image collection device 220 can be subjectedto parameter calibration, so that the location of the gaze point F ofthe user eye with respect to the user substantially can be considered asthe location of the gaze point F with respect to the image collectiondevice 220; and in this case, without a need of adjusting the shootingangle of the image collection device and additionally calculating thedistance of the gaze point F with respect to the image collection device220, the focusing of the image collection device can be adjusteddirectly according to the location of the gaze point to obtain a firstimage needed by the user.

In other possible implementations of the embodiments of the presentapplication, the image collection device can be arranged on a locationoutside the wearable near-to-eye apparatus 210, for example, the imagecollection device can be arranged on other wearable apparatuses (forexample, a smart watch) of a user; or can also be arranged on ahand-held apparatus (such as a cell phone, a tablet computer and otherapparatuses with an image collection module) of a user and so on; or theimage collection device is a photo camera, a video camera and otherapparatuses special for image collection; or can also be a monitoringapparatus (such as a monitoring camera, etc.) fixedly arranged in ascene.

In this implementation, the image collection device 220 comprises:

a focusing module 221 for adjusting the focusing point location of theimage collection device according to the location information andposture information about the image collection device, the locationinformation and posture information about a user, and the location ofthe gaze point F with respect to the user. In this implementation, byacquiring the location information (including the location informationabout the image collection device in a scene or the location informationthereof with respect to a user) and the posture information (includingthe shooting optical axis direction and other posture information) aboutthe image collection device, and then after the location informationabout the gaze point F with respect to the image collection device 220has been obtained by calculating according to the location informationabout the gaze point F with respect to the user, focusing and imagecollection are performed.

In this implementation, the system further comprises:

a user location information acquisition device 230 for acquiringlocation information about a user;

a user posture information acquisition device 240 for acquiring postureinformation about the user;

a collection-device location information acquisition device 250 foracquiring location information about each image collection device; and

a collection-device posture information acquisition device 260 foracquiring posture information about each image collection device.

In this case:

in a possible implementation, the user location information acquisitiondevice 230 can be a locating module, which collects location informationabout a user by means of an outdoor or indoor locating module. Inanother possible implementation, the user location informationacquisition device 230 can also be a communication module for receivinglocation information about a user transmitted from the external.

In a possible implementation, the user posture information acquisitiondevice 240 can be a posture information sensing module for collectingposture information about the user, posture information about the userincluding the face orientation of the user and so on. In anotherpossible implementation, the user posture information acquisition device240 can also be a communication module for receiving posture informationabout the user transmitted from the external.

In a possible implementation, the collection-device location informationacquisition device 250 can be the locating module or communicationmodule described above, which acquires location information about animage collection device by way of collecting or receiving same; or, fora monitoring apparatus (such as a monitoring camera) fixed at a specificlocation, since the location thereof is fixed, the location informationabout the image collection device is acquired by way of pre-storing thelocation information about the image collection device 220 in a localstorage apparatus, and the collection-device location informationacquisition device 250 reading same from the local storage apparatus.

In a possible implementation, the collection-device posture informationacquisition device 260 can be a posture information sensing module forcollecting posture information about an image collection device, theposture information about the image collection device including theoptical axis direction of the image collection device and so on. Inanother possible implementation, the image-collection-device postureinformation acquisition device 260 can also be a communication modulefor receiving posture information about the image collection devicetransmitted from the external. Furthermore, in other possibleimplementations, the image collection device may be fixed still, withthe posture thereof being unadjustable; in this case, the postureinformation about the image collection device may also be read from alocal storage apparatus.

As shown in FIG. 3, in a possible implementation of the embodiments ofthe present application, the system comprises a plurality of imagecollection devices 310. The plurality of image collection devices 310may include a plurality of forms of image collection devices 310, forexample, comprising one or more of an image collection device which isarranged together with the gaze point detection device on a wearablenear-to-eye apparatus 210, an image collection device on a cell phoneand other portable intelligent apparatuses, a photo camera, a videocamera and the like arranged at a required location by a user, amonitoring apparatus fixedly arranged in a scene, etc.

Generally, the plurality of image collection devices 310 are disposedseparately, so that they can cooperatively collect an image of an objecton which a user fixes at a plurality of angles.

In this case, in a possible implementation of the embodiments of thepresent application, the system further comprises:

an association device 320 for selecting at least one image collectiondevice 310 to associate same with the gaze point detection device 270;and

the image collection device 310 focusing and collecting a first imageaccording to the location of the gaze point with respect to the userwhen it is associated with the gaze point detection device 270.

In this implementation, the association device 320 can select an imagecollection device 310 according to an external instruction (for example,an instruction sent by a user) to associate same with the gaze pointdetection device 270 or release the association thereof; and can alsoautomatically perform the association or association release operation,for example, automatically detecting an image collection device 310 in ascene, and automatically associating same if detected, or can alsoautomatically judge whether an image collection device 310 in a scenesatisfies a set association condition (for example, the associationcondition can be set according to the location and/or performance of theimage collection device 310), and automatically associate the imagecollection device 310 if satisfied.

As shown in FIG. 4a , in a possible implementation of the embodiments ofthe present application, the gaze point detection device 500 comprises afundus image collection module 510, an adjustable lens module 520 and animage processing module 530 described in the embodiments above.

The gaze point detection device 500 in the embodiments of the presentapplication, by processing an image of the fundus of an eye, obtains anoptical parameter of the eye when the fundus image collection module hasobtained at least one second image that satisfies at least one definedclarity criterion and at least one imaging parameter of the optical pathbetween the eye and the fundus image collection module, so that thelocation of the gaze point of the eye can be calculated, providing abasis for further realizing a self-adaptive operation for eyes.

The image presented at the “fundus” herein mainly is an image presentedon the retina, which can be an image of the fundus itself, or can alsobe an image of other objects projected to the fundus (such as the lightspot pattern mentioned below). The eye herein can be a human eye, andcan also be an eye of other animals.

In a possible implementation of the embodiments of the presentapplication, the fundus image collection module 510 can be a microcamera, and in another possible implementation of the embodiments of thepresent application, the fundus image collection module 510 can alsodirectly use a photosensitive imaging device, such as a CCD device or aCMOS device, etc.

In a possible implementation of the embodiments of the presentapplication, the adjustable lens module 520 is located on the opticalpath between an eye and the fundus image collection module 510, with thefocal length thereof being adjustable and/or the location thereof in theoptical path being adjustable. The equivalent focal length of an opticalsystem between an eye and the fundus image collection module 510 isenabled to be adjustable by means of the adjustable lens module 520, andby adjusting the adjustable lens module 520, the fundus image collectionmodule 510 is enabled to obtain a clearest second image of the fundus insome location or state of the adjustable lens module 520. In thisimplementation, the adjustable lens module 520 can be adjustedcontinuously in real time during detection.

In a possible implementation of the embodiments of the presentapplication, the adjustable lens module 520 can be a focal-lengthadjustable lens for completing the adjustment of the focal length of itsown by adjusting the refractive index and/or shape of its own. Inparticular, 1) the focal length is adjusted by adjusting the curvatureof at least one surface of the focal-length adjustable lens, forexample, adjusting the curvature of the focal-length adjustable lens byincreasing or decreasing a liquid medium in a cavity formed by adouble-layer transparent layer; 2) the focal length is adjusted bychanging the refractive index of the focal-length adjustable lens, forexample, filling a specific liquid crystal medium in the focal-lengthadjustable lens, and adjusting the arrangement mode of the liquidcrystal medium by adjusting the voltage of a corresponding electrode ofthe liquid crystal medium, to thereby change the refractive index of thefocal-length adjustable lens.

The embodiments of the present application adopt the focal-lengthadjustable lens as the adjustable lens module 520, and therefore,compared with US Patent Publication No. U.S. Pat. No. 7,298,414 B2 inwhich an image reflected by an eye is obtained by moving the location ofa lens and/or an image sensor, the gaze point detection device 500 inthe embodiments of the present application is of a smaller and morecompact structure, and is suitable to be applied to a wearableapparatus.

In another possible implementation of the embodiments of the presentapplication, the adjustable lens module 520 comprises: a lens setcomposed of a plurality of lenses for completing the adjustment of thefocal length of the lens set itself by adjusting the relative locationsof the lenses in the lens set. The lens set may also comprise a lenswith adjustable focal length and other imaging parameters.

In addition to the above-mentioned two ways of changing the optical pathparameters of the system by adjusting the characteristics of theadjustable lens module 520 itself, the optical path parameters of thesystem can be changed by adjusting the location of the adjustable lensmodule 520 on the optical path.

In a possible implementation of the embodiments of the presentapplication, in order not to affect the viewing experience of a user toan observed object, and in order to enable a module to be portablyapplied to a wearable apparatus, the gaze point detection device 500 mayalso comprise: a beam splitting module 550 for forming a light transferpath between an eye and the observed object and between the eye and thefundus image collection module 510. In this way, an optical path can befolded to reduce the volume of a module, and at the same time, othervisual experiences of a user are not affected as much as possible.

In this implementation, the beam splitting module 550 may comprise: afirst beam splitting unit which is located between an eye and anobserved object, and used for transmitting the light from the observedobject to the eye and transferring the light from the eye to a fundusimage collection module.

The first beam splitting unit can be a beam splitter, a beam splittingoptical waveguide (including an optical fiber) or other suitable beamsplitting apparatuses.

In a possible implementation of the embodiments of the presentapplication, the image processing module 530 of the module comprises anoptical path calibration module for calibrating the optical path of thesystem, for example, aligning and calibrating the optical axis of theoptical path, etc., to ensure the precision of measurement.

In a possible implementation of the embodiments of the presentapplication, the image processing module 530 comprises:

an image analysis unit 531 for analyzing at least one second imageobtained by the fundus image collection module to find out a clearestsecond image; and

a parameter calculation unit 532 for calculating an optical parameter ofan eye and the location of the gaze point of the eye with respect to auser according to the clearest image and the known imaging parameters ofthe module when the clearest image is obtained.

In this implementation, the fundus image collection module 510 isenabled to obtain a clearest image by means of the adjustable lensmodule 520, but it needs to find out the clearest image by means of theimage analysis unit 531, and then, the optical parameter of an eye canbe calculated out according to the clearest image and the known opticalpath parameters of the module. The optical parameter of an eye hereincan include the optical axis direction of the eye.

In a possible implementation of the embodiments of the presentapplication, the gaze point detection device 500 may also comprise: aprojection module 540 for projecting a light spot to the fundus. In apossible implementation, the function of the projection module can berealized by means of a micro projector.

The light spot projected herein can have no specific pattern and bemerely used for lighting the fundus.

In an implementation of the embodiments of the present application, theprojected light spot can include a pattern with abundant features. Theabundant features of a pattern can facilitate the detection and enhancethe detection precision. What is shown in FIG. 4b is an exemplarydiagram of a light spot pattern 550, which pattern can be formed by alight spot pattern generator, such as frosted glass; and what is shownin FIG. 4c is an image of the fundus shot when the light spot pattern550 is projected thereon.

In order not to affect the normal viewing of an eye, the light spot canbe an infrared light spot which is invisible to the eye.

In this case, in order to reduce the disturbance of other spectra:

an emergent surface of the projection module 540 can be provided with aneye-invisible light transmission filter.

An incident surface of the fundus image collection module 510 isprovided with an eye-invisible light transmission filter.

In a possible implementation of the embodiments of the presentapplication, the image processing module 530 may also comprise:

a projection control unit 534 for controlling the brightness of thelight spot projected by the projection module according to a resultobtained by an image analysis unit.

For example, the projection control unit 534 can self-adaptively adjustthe brightness according to the characteristics of an image obtained bythe fundus image collection module 510. The characteristics of an imageherein include the contrast of image features, texture features, etc.

Here, a special circumstance of controlling the brightness of the lightspot projected by the projection module is to turn on or turn off theprojection module, for example, the projection module can be turned offperiodically when a user continuously fixes on one point; and when thefundus of a user is bright enough, a light emitting source can be turnedoff and the distance from the gaze point of the sight line of an eye tothe eye can be detected only using the information about the fundus.

Furthermore, the projection control unit 534 can also control thebrightness of the light spot projected by the projection moduleaccording to the ambient light.

In a possible implementation of the embodiments of the presentapplication, the image processing module 530 may also comprise: an imagecalibration unit 533 for calibrating a fundus image to obtain at leastone reference image corresponding to the image presented at the fundus.

The image analysis unit 531 compares at least one second image obtainedby the fundus image collection module 530 with the reference image andcalculates same to obtain the clearest second image. Here, the clearestsecond image can be an image obtained the difference between which andthe reference image is minimum. In this implementation, the differencebetween the second image obtained and the reference image is calculatedby means of an existing image processing algorithm, for example, using aclassical phase difference automatic focusing algorithm.

In a possible implementation of the embodiments of the presentapplication, the parameter calculation unit 532 may comprise:

an eye optical axis direction determination subunit 5321 for obtainingthe optical axis direction of an eye according to the features of theeye when the clearest second image is obtained.

The features of an eye herein can be acquired from the clearest secondimage, or can also be acquired otherwise. The optical axis direction ofan eye represents the gaze direction of the sight line of the eye.

In a possible implementation of the embodiments of the presentapplication, the eye optical axis direction determination subunit 5321may comprise: a first determination subunit for obtaining the opticalaxis direction of an eye according to the features of the fundus whenthe clearest second image is obtained. Compared with obtaining theoptical axis direction of an eye by means of the features of the pupiland the eyeball surface, the accuracy of obtaining the optical axisdirection of an eye by means of the features of the fundus is higher.

When a light spot pattern is projected to the fundus, the size of thelight spot pattern may be larger than a visible region of the fundus orsmaller than that, wherein:

when the area of the light spot pattern is smaller than or equal to thatof the visible region of the fundus, the optical axis direction of aneye can be determined by detecting the location of the light spotpattern on a second image with respect to the fundus using a classicalfeature point matching algorithm (for example, the scale invariantfeature transform algorithm (SIFT));

when the area of the light spot pattern is greater than or equal to thatof the visible region of the fundus, the optical axis direction of aneye can be determined by means of the location of the light spot patternon the second image obtained with respect to an original light spotpattern (obtained by an image calibration unit) so as to determine thedirection of sight line of a user.

In a possible implementation of the embodiments of the presentapplication, the eye optical axis direction determination subunit 5321comprises: a second determination subunit for obtaining the optical axisdirection of an eye according to the features of the eye pupil when theclearest second image is obtained. The features of the eye pupil hereincan be acquired from the clearest second image, and can also be acquiredotherwise. The obtainment of the optical axis direction of an eye bymeans of the features of the eye pupil is an existing technology, whichwill not be described here.

In a possible implementation of the embodiments of the presentapplication, the image processing module 530 may also comprise: an eyeoptical axis direction calibration unit 535 for calibrating the opticalaxis direction of an eye to determine the optical axis direction of theeye more accurately.

In this implementation, the known imaging parameters of the moduleinclude a fixed imaging parameter and a real-time imaging parameter,wherein the real-time imaging parameter is the parameter informationabout the adjustable lens unit when a clearest second image is acquired,and the parameter information can be obtained by recording in real timewhen the clearest second image is acquired.

After the known imaging parameters of an optical path system between aneye and a fundus image collection module has been obtained, the distancefrom the gaze point of the eye to the eye can be calculated out, whichis in particular as follows:

what is shown in FIG. 5a is a schematic diagram of eye imaging, andequation (1) can be obtained from FIG. 5a in combination with the lensimaging equation in the classical optical theory:

$\begin{matrix}{{\frac{1}{d_{o}} + \frac{1}{d_{e}}} = \frac{1}{f_{e}}} & (1)\end{matrix}$

where d_(o) and d_(e) are the distance from an observed object 5010 ofan eye to an eye equivalent lens 5030 and the distance from a real image5020 on the retina to the eye equivalent lens 5030 respectively, f_(e)is the equivalent focal length of the eye equivalent lens 5030, and X isthe optical axis direction of the eye (i.e., the optical axis of thesight line).

What is shown in FIG. 5b is a schematic diagram where the distance fromthe gaze point of an eye to the eye is obtained according to the knownoptical parameters of the module and the optical parameters of the eye;in FIG. 5b , a light spot 5040 will form a virtual image (not shown inFIG. 5b ) via an adjustable lens module 520, and by assuming thedistance of the virtual image from the lens to be X (not shown in FIG.5b ), in combination with the equation (1), the following set ofequations can be obtained:

$\begin{matrix}\{ \begin{matrix}{{\frac{1}{d_{p}} - \frac{1}{x}} = \frac{1}{f_{p}}} \\{{\frac{1}{d_{1} + x} + \frac{1}{d_{e}}} = \frac{1}{f_{e}}}\end{matrix}  & (2)\end{matrix}$

where d_(p) is the optical equivalent distance from the light spot 5040to the adjustable lens module 520, d_(l) is the optical equivalentdistance from the adjustable lens module 520 to the eye equivalent lens5030, is the focal length value of the adjustable lens module 520, andd_(l) is the distance from the eye equivalent lens 5030 to theadjustable lens module 520.

The distance d_(o) from the observed object 5010 (the gaze point of theeye) to the eye equivalent lens 5030 can be obtained as shown inequation (3) from (1) and (2):

$\begin{matrix}{d_{o} = {d_{i} + \frac{d_{p} \cdot f_{p}}{f_{p} - d_{p}}}} & (3)\end{matrix}$

According to the distance from the observed object 5010 to the eyecalculated above, and the optical axis direction of the eye which can beobtained as a result of the preceding description, the locationinformation about the gaze point of the eye with respect to a user canbe obtained easily, providing a basis for subsequent further interactionassociated with the eye.

What is shown in FIG. 6 is the embodiment of applying a gaze pointdetection device 600 in a possible implementation of the embodiments ofthe present application to a pair of spectacles 400, which comprises thecontents described in the implementation as shown in FIG. 4 a, and is inparticular as follows: it can be seen from FIG. 6 that in thisimplementation, the gaze point detection device 600 of thisimplementation is integrated to the right side of the spectacles 400,comprising:

a micro camera 610 which functions the same as the fundus imagecollection module described in the implementation of FIG. 4a , and isarranged at the outer right side of an eye 200 in order not to affectthe sight line of a user normally viewing an object;

a first beam splitter 620 which functions the same as the first beamsplitting unit described in the implementation of FIG. 4a , is arrangedwith a certain tilt angle at the intersection point of the gazedirection of the eye 200 and the incident direction of the camera 610,and transmits the light entering into the eye 200 from an observedobject and reflects the light from the eye to the camera 610; and

a focal-length adjustable lens 630 which functions the same as thefocal-length adjustable lens described in the implementation of FIG. 4a, is located between the first beam splitter 620 and the camera 610, andadjusts the focal length value in real time, so that the camera 610 canshoot a clearest second image of the fundus at some focal length value.

In this implementation, the image processing module is not shown in FIG.6, and functions the same as the image processing module as shown inFIG. 4 a.

Since the brightness of the fundus is not enough under normalcircumstances, the fundus had better be illuminated, and in thisimplementation, the fundus is illuminated by a light emitting source640. In order not to affect the user experience, the light emittingsource 640 herein can be an eye-invisible light emitting source, andfurther, can be a near-infrared light emitting source which does notmuch affect the eye 200 and to which the camera 610 is relativelysensitive.

In this implementation, the light emitting source 640 is located at theouter side of the right side of a spectacle frame, so it needs a secondbeam splitter 650 together with the first beam splitter 620 to completethe transferring of the light emitted by the light emitting source 640to the fundus. In this implementation, the second beam splitter 650 isalso located in front of the incident surface of the camera 610, so itneeds to transmit the light from the fundus to the camera 610.

It can be seen that in this implementation, in order to enhance the userexperience and enhance the collection definition of the camera 610, thefirst beam splitter 620 can have the characteristics of highreflectivity to infrared and high transmissivity to visible light. Forexample, an infrared reflective film can be arranged at the side of thefirst beam splitter 620 facing the eye 200 to realize thecharacteristics described above.

It can be seen from FIG. 6 that since in this implementation, the eyegaze point detection device 600 is located at the side of the lens ofthe spectacles 400 away from the eye 200, the lens can be considered asa part of the eye 200 during the calculation of the optical parametersof the eye, without a need of knowing the optical characteristics of thelens.

In other implementations of the embodiments of the present application,the eye gaze point detection device 600 may be located at the side ofthe lens of the spectacles 400 close to the eye 200, and then, it needsto obtain the optical characteristic parameters of the lens in advanceand take the affecting factor of the lens into account when the distancefrom the gaze point to the eye of a user is being calculated.

The light emitted by the light emitting source is reflected by thesecond beam splitter 650, transmitted by the focal-length adjustablelens 630 and reflected by the first beam splitter 620, then transmitsthrough the lens of the spectacles 400 to enter into the eye of a user,and finally arrives at the retina of the fundus; and the camera 610shoots an image of the fundus through the pupil of the eye 200 via anoptical path composed of the first beam splitter 620, the focal-lengthadjustable lens 630 and the second beam splitter 650.

What is shown in FIG. 7 is a structural schematic diagram of a gazepoint detection device 700 of another implementation of the embodimentsof the present application. It can be seen from FIG. 7 that thisimplementation is similar to the implementation shown in FIG. 6,comprising a micro camera 710, a second beam splitter 720 and afocal-length adjustable lens 730, except that the projection module 740in this implementation is a projection module 740 for projecting a lightspot pattern, and the first beam splitter in the implementation of FIG.6 is replaced with a curved beam splitter 750 as a curved beam splittingmodule.

The curved beam splitter 750 is adopted here to transfer at least oneimage presented at the fundus to the fundus image collection module, andwherein the at least one image respectively correspond to at least onelocation of the pupil associated with different optical axis directionsof the eye. In this way, the camera can shoot the mixed and superimposedimages formed of various angles of the eyeball; however, since only thefundus part through the pupil can clearly focus on the camera, whileother parts will defocus and thereby fail at clear focus, the formedimage of the fundus part will not be disturbed severely, and thefeatures of the fundus part can still be detected out. Therefore,compared with the implementation shown in FIG. 6, this implementationcan obtain an image of the fundus well in different gaze directions ofan eye, so that the eye gaze point detection device of thisimplementation has wider scope of application and higher detectionprecision.

In a possible embodiment of the embodiments of the present application,the embodiments of the present application are described further with animage collection system only having one image collection device:

in this embodiment, the gaze point detection device is arranged on aspectacle apparatus, and the image collection device is also arranged onthe spectacle apparatus; in this case, it can be thought that the firstimage collected by the image collection device is substantially theimage seen by a user.

According to the implementations described in the embodiments of FIG. 4ato FIG. 7, the location information about a gaze point of a user withrespect to the user and the orientation and distance of the gaze pointwith respect to the user are obtained in real time by means of the gazepoint detection device;

the focal length and other parameters of the image collection device areadjusted according to the distance of the gaze point with respect to theuser; and after the parameters of the image collection device have beenadjusted successfully, a shutter is triggered directly to collect afirst image. Or in other embodiments, the image collection device istriggered via an external instruction to collect a first image; forexample, the trigger instruction is transmitted by means of a wink of auser.

In another possible embodiment of the embodiments of the presentapplication, the embodiments of the present application are describedfurther with an image collection system having a plurality of imagecollection devices disposed separately (the image collection system asshown in FIG. 3):

in this embodiment, in addition to a gaze point detection device andimage collection devices, the image collection system further comprisesan association device. In this embodiment, as an example, the gaze pointdetection device is arranged on a spectacle apparatus, and theassociation device is also arranged on the spectacle apparatus, and thespectacle apparatus herein can be an intelligent spectacle apparatus.

By means of the association device, a plurality of image collectiondevices are selected simultaneously or in turn to be associated with thegaze point detection device. For example, some image collection deviceis associated by means of an action instruction such as a wink and soon, and the intelligent spectacle apparatus may have a display interfaceor an audible prompt to prompt a user that some or a plurality of imagecollection device(s) has(have) been associated;

after the location of the gaze point of the user with respect to theuser has been obtained by means of the gaze point detection device, eachimage collection device having been associated, according to thelocation and posture information about its own and the location andposture information about the user, calculates out a parameteradjustment value of its own, such as an adjustment value of shootingangle, an adjustment value of focusing parameters, etc., and focuses onan object corresponding to the gaze point according to these parameteradjustment values (in which process, the operating state, such as “beingcalculating”, “being focusing”, “focusing succeeded” and so on, of theimage collection device can be fed back to the intelligent spectacleapparatus and thereby transmitted to the user);

after having succeeded in focusing, the image collection device canautomatically collect a first image, or after the image collectiondevice has succeeded in focusing, the user sends out an image collectiontrigger instruction (for example, he/she can send out the instruction bymeans of the intelligent spectacle apparatus), to control the imagecollection device to collect a first image;

the first image (including a picture or a video) obtained by the imagecollection device can be exhibited to the user by means of theintelligent spectacle apparatus, for the user to perform a furtheroperation (such as editing or selecting);

the user sends out an instruction again to the association device torelease the association with the image collection device.

As shown in FIG. 8, the embodiments of the present application alsoprovide an image collection method, comprising:

S110: a gaze point detection step of detecting the location of a gazepoint of a user with respect to the user; and

S120: an image collection step of focusing and collecting at least onefirst image according to the location of the gaze point with respect tothe user;

wherein the step S110 comprises:

a fundus image collection step of collecting at least one second imageof a fundus of the user;

an imaging parameter adjustment step of adjusting at least one imagingparameter of an optical path between the collection location of the atleast one second image and an eye of the user until at least one secondimage that satisfies at least one defined clarity criterion iscollected; and

an image processing step of processing the at least one second imagethat satisfies at least one defined clarity criterion, obtaining the atleast one imaging parameter corresponding to the at least one secondimage that satisfies at least one defined clarity criterion and theoptical axis direction of the eye, and calculating the location of thegaze point of the user with respect to the user.

The present application obtains the optical axis direction of an eye ofa user and the distance from a gaze point to the eye by means of thegaze point detection step, and thereby can calculate out the location ofthe gaze point of the user with respect to the user, and then focusesfor image collection according to the location of the gaze point, sothat the image collection step can rapidly and accurately focus on theinterested object of the user and collect a first image, enhancing theuser experience.

In a possible implementation of the embodiments of the presentapplication, the imaging parameter adjustment step may comprise:

adjusting the focal length of at least one adjustable lens modulelocated on the optical path between the collection location of thesecond image and an eye of the user.

In a possible implementation of the embodiments of the presentapplication, the gaze point detection step may also comprise:

transferring a at least one image presented at the fundus to thecollection location of the at least one second image, and wherein the atleast one image respectively correspond to at least one location of thepupil associated with different optical axis directions of the eye.

In a possible implementation of the embodiments of the presentapplication, the gaze point detection step may also comprise:

projecting at least one light spot pattern to the fundus.

The particular implementation of the gaze point detection step can referto the corresponding description in the gaze point detection step shownin FIG. 4a to FIG. 7, which will not be described in this embodiment.

In a possible implementation of the embodiments of the presentapplication, in order to cooperatively focus on an object on which auser fixes at a plurality of angles, the image collection step maycomprise: performing the focusing and collecting at least one firstimage by a plurality of image collection devices.

In a possible implementation of the embodiments of the presentapplication, the plurality of image collection devices can be disposedseparately.

In a possible implementation of the embodiments of the presentapplication, when an image collection device and a user are not in oneand the same location, the image collection step may comprise:

adjusting the focusing point location of the image collection deviceaccording to the location information and posture information about thecollection device of the second image and the user, and the location ofthe gaze point with respect to the user.

In a possible implementation of the embodiments of the presentapplication, the method may also comprise: acquiring the locationinformation about the user.

In a possible implementation of the embodiments of the presentapplication, the method may also comprise: acquiring the postureinformation about the user.

In a possible implementation of the embodiments of the presentapplication, the method may also comprise: acquiring the locationinformation about each image collection device.

In a possible implementation of the embodiments of the presentapplication, the method may also comprise: acquiring the postureinformation about each image collection device.

In a possible implementation of the embodiments of the presentapplication, the method may also comprise:

determining at least one image collection device in the plurality ofimage collection devices according to the location of the gaze point ofthe user with respect to the user; and

-   -   in the image collection step, performing the focusing and        collecting a first image by means of the at least one image        collection device determined, according to the location of the        gaze point with respect to the user.

The implementation of each step described above is the same as thedescription of the functions of corresponding module or unit in theembodiments of the device described above, which will not be describedhere.

It should be understood that in various embodiments of the presentinvention, the serial number of each procedure described above does notmean an execution sequence, and the execution sequence of each procedureshould be determined according to the function and internal logicthereof, and should not be any limitation on the implementationprocedure of the embodiments of the present invention.

It can be appreciated by those skilled in the art that each exemplaryunit and method step described with reference to the embodimentsdisclosed in this document can be realized by electronic hardware or acombination of computer software and electronic hardware. Whether thesefunctions are executed in a hardware mode or a software mode depends onthe specific applications and design constraint conditions of thetechnical solution. Those professional technicians can use differentmethods to realize the functions described with respect to each specificapplication, but this realization shall not be considered beyond thescope of the present application.

If the function is realized in the form of a software functional unitand is sold or used as an independent product, it can be stored in acomputer-readable storage medium. Based on such understanding, thetechnical solution of the present application essentially or the partwhich contributes to the prior art or a part of the technical solutioncan be embodied in the form of a software product, and the computersoftware product is stored in a storage medium, and comprises severalinstructions for enabling a computer apparatus (which can be a personalcomputer, a server, or a network apparatus, etc.) to execute all or somesteps of the method described in each embodiment of the presentapplication. The preceding storage medium comprises various medium whichcan store a program code, such as a USB disk, a mobile hard disk, aread-only memory (ROM), a random access memory (RAM), a magnetic disc ora compact disk, etc.

The above implementations are only used to describe the presentapplication, without limiting the present application; variousalterations and variants can be made by those skilled in the related artwithout departing from the spirit and scope of the present application,so all equivalent technical solutions also belong to the scope of thepresent application, and the scope of patent protection of the presentapplication should be defined by claims.

1. An image collection system, comprising: a gaze point detection deviceconfigured to detect the location of a gaze point of a user with respectto the user; and at least one image collection device configured tofocus and collect at least one first image according to the location ofthe gaze point with respect to the user; wherein the gaze pointdetection device comprises: a fundus image collection module configuredto collect at least one second image of a fundus of the user; at leastone adjustable lens module configured to adjust at least one imagingparameter of an optical path between the fundus image collection moduleand an eye of the user until the fundus image collection module obtainsat least a subset of the at least one second image that satisfies atleast one defined clarity criterion; and an image processing moduleconfigured to process at least the subset of the at least one secondimage that satisfies the at least one defined clarity criterion, obtainthe at least one imaging parameter corresponding to at least the subsetof the at least one second image that satisfies the at least one definedclarity criterion and an optical axis direction of the eye, anddetermine the location of the gaze point of the user with respect to theuser.
 2. The image collection system of claim 1, wherein a focal lengthof the at least one adjustable lens module is adjustable.
 3. The imagecollection system of claim 1, wherein the gaze point detection devicefurther comprises: a curved beam splitting module configured to transferat least one third image presented at the fundus to the fundus imagecollection module, and wherein the at least one third image respectivelycorrespond to at least one location of the pupil associated withdifferent optical axis directions of the eye.
 4. The image collectionsystem of claim 1, wherein the gaze point detection device furthercomprises: a projection module configured to project at least one lightspot pattern to the fundus.
 5. The image collection system of claim 1,further comprising a wearable near-to-eye apparatus, wherein the gazepoint detection device is arranged on the wearable near-to-eyeapparatus.
 6. The image collection system of claim 5, wherein the atleast one image collection device is arranged on the wearablenear-to-eye apparatus.
 7. The image collection system of claim 5,wherein the wearable near-to-eye apparatus is a pair of spectacles. 8.The image collection system of claim 1, wherein the at least one imagecollection device is a plurality of image collection devices.
 9. Theimage collection system of claim 8, wherein the plurality of imagecollection devices are disposed separately.
 10. The image collectionsystem of claim 8, wherein each image collection device in the at leastone image collection device comprises: a focusing module configured toadjust the focusing point location of at least one image collectiondevice to which the focusing module belongs, according to locationinformation about the at least one image collection device to which thefocusing module belongs, posture information about the at least oneimage collection device to which the focusing module belongs, locationinformation about the user, user posture information about the user andthe location of the gaze point with respect to the user.
 11. The imagecollection system of claim 10, further comprising: at least one userlocation information acquisition device for acquiring the locationinformation about the user.
 12. The image collection system of claim 10,further comprising: at least one user posture information acquisitiondevice configured to acquire the user posture information about theuser.
 13. The image collection system of claim 10, further comprising:at least one collection-device location information acquisition deviceconfigured to acquire the location information about the at least oneimage collection device.
 14. The image collection system of claim 10,further comprising: at least one collection-device posture informationacquisition device configured to acquire the posture information aboutthe at least one image collection device.
 15. The image collectionsystem of claim 8, further comprising: an association device configuredto select at least one image collection device from the plurality ofimage collection devices to associate the at least one image collectiondevice with the gaze point detection device; and the at least one imagecollection device in the plurality of image collection devicesconfigured to focus and collect the at least one first image accordingto the location of the gaze point with respect to the user.
 16. Amethod, comprising: detecting a location of a gaze point of a user withrespect to the user; and focusing and collecting at least one firstimage according to the location of the gaze point with respect to theuser, wherein the detecting comprises: collecting at least one secondimage of a fundus of the user; adjusting at least one imaging parameterof an optical path between a collection location of the at least onesecond image and an eye of the user, until at least a subset of the atleast one second image that satisfies at least one defined claritycriterion is obtained during the collecting the at least one secondimage of the fundus of the user; processing at least the subset of theat least one second image that satisfies the at least one definedclarity criterion; obtaining the at least one imaging parametercorresponding to at least the subset of the at least one second imagethat satisfies the at least one defined clarity criterion and an opticalaxis direction of the eye; and determining the location of the gazepoint of the user with respect to the user.
 17. The method of claim 16,wherein the adjusting the at least one imaging parameter of comprises:adjusting a focal length of at least one adjustable lens module locatedon the optical path between the collection location of the at least onesecond image and the eye of the user.
 18. The method of claim 16,wherein the detecting the location of the gaze point comprises:transferring at least one third image presented at the fundus to thecollection location of the at least one second image, and wherein the atleast one third image respectively correspond to at least one locationof the pupil associated with different optical axis directions of theeye.
 19. The method of claim 17, wherein the detecting the location ofthe gaze point comprises: projecting at least one light spot pattern tothe fundus.
 20. The method claim 16, wherein the focusing and thecollecting of the at least one first image according to the location ofthe gaze point with respect to the user comprises: focusing andcollecting the at least one first image using a plurality of imagecollection devices.
 21. The method of claim 20, wherein the plurality ofimage collection devices are disposed separately.
 22. The method ofclaim 20, wherein the focusing and the collecting of the the at leastone first image according to the location of the gaze point with respectto the user further comprises: for each image collection device in theplurality of image collection devices, adjusting the focusing pointlocation of the each image collection device according to locationinformation about the each image collection device, posture informationabout the each image collection device, location information about theuser, user posture information about the user and the location of thegaze point with respect to the user.
 23. The method of claim 22, furthercomprising: acquiring the location information about the user.
 24. Themethod of claim 22, further comprising: acquiring the user postureinformation about the user.
 25. The method of claim 22, furthercomprising: acquiring the location information about the plurality ofimage collection devices.
 26. The method of claim 22, furthercomprising: acquiring the posture information about the plurality ofimage collection devices.
 27. The method of claim 20, furthercomprising: determining at least one image collection device in theplurality of image collection devices according to the location of thegaze point of the user with respect to the user, wherein the focusingand the collecting of the at least one first image comprises focusingand collecting the at least one first image by the at least one imagecollection device according to the location of the gaze point withrespect to the user.
 28. A computer-readable storage device, comprisingat least one executable instruction, which, in response to execution,causes a device comprising a processor to perform operations,comprising: detecting a location of a gaze point of a user with respectto the user; and focusing and collecting at least one first imageaccording to the location of the gaze point with respect to the user;wherein the detecting the location of the gaze point comprises:collecting at least one second image of a fundus of the user; adjustingat least one imaging parameter of an optical path between a collectionlocation of the at least one second image and an eye of the user, untilthe at least one second image that satisfies at least one definedclarity criterion is obtained during the collecting of the at least onesecond image of the fundus of the user; processing the at least onesecond image that satisfies the at least one defined clarity criterion;obtaining the at least one imaging parameter corresponding to the atleast one second image that satisfies the at least one defined claritycriterion and an optical axis direction of the eye; and calculating thelocation of the gaze point of the user with respect to the user.
 29. Animage collection system, characterized by comprising a processing deviceand a memory, wherein the memory stores at least one executableinstruction, the processing device is connected to the memory via acommunication bus, and when the image collection system executes, theprocessing device executes the at least one executable instructionstored in the memory, so that the image collection system executesoperations, comprising: detecting a location of a gaze point of a userwith respect to the user; and focusing and collecting at least one firstimage according to the location of the gaze point with respect to theuser; wherein the detecting the location of the gaze point comprises:collecting at least one second image of a fundus of the user; adjustingat least one imaging parameter of an optical path between a collectionlocation of the at least one second image and an eye of the user, untilthe at least one second image that satisfies at least one definedclarity criterion is obtained during the collecting of the at least onesecond image of the fundus of the user; and processing the at least onesecond image that satisfies the at least one defined clarity criterion;obtaining the at least one imaging parameter corresponding to the atleast one second image that satisfies the at least one defined claritycriterion and an optical axis direction of the eye; and calculating thelocation of the gaze point of the user with respect to the user.